Minimizing Thermal Impacts of Hydropower using Reinforcement Learning

Operational policy for hydropower dams must balance energy generation goals with environmental impacts under federal regulation, such as the Endangered Species and the Clean Water Acts. Multiple impoundments, reservoirs, outlet structures, and stochastic weather patterns complicate decision making to minimize impacts on downstream ecosystems. Currently, hydropower management to meet water quality goals is largely based on seasonal rules-ofthumb. Non-optimal management combined with recent, climatically novel heat waves has led to high die-off rates for fish, who cannot tolerate high water temperatures. For example, high temperatures in spring and summer of 2015 caused a die-off of over 95% of a sockeye salmon cohort returning to the Snake River in the state of Washington[7]. Water undergoes significant increases in density as its temperature drops. In a reservoir, this property can create stratification, sequestering cool water low in the vertical profile while warmer water sits at the top. Dams often have two or more outlets from top to bottom through which water can be released, with the lower outlets passing cooler water and the upper outlets passing warmer water. However, impounded waters heat over the course of the year, undergoing complex nonlinear meteorological and hydrological forcing. Thus, maintaining available volumes of water at appropriate temperature for release during the warmest months requires nontrivial planning. We demonstrate the applicability of Q-learning to optimize policy for dam operations. We computationally simulate the hydrodynamics of a hydropower dam, and apply Q-learning to identify optimal actions day by day. We report here on two separate learning objectives that optimize two subgoals of dam operations: keeping water elevation levels within operational limits, and minimizing the temperature increase across the reservoir.